Skip to content

Feat: monitor automations opt out - #2702

Open
BelhsanHmida wants to merge 48 commits into
mainfrom
feat/2519-monitor-automations-opt-out
Open

BelhsanHmida wants to merge 48 commits into
mainfrom
feat/2519-monitor-automations-opt-out

Conversation

@BelhsanHmida

@BelhsanHmida BelhsanHmida commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

Lets an automation opt out of flexmeasures monitor automations, for instance an experiment whose failures nobody needs to hear about.

  • A new monitored field on automations, true by default and for every existing automation.
  • Settable wherever automations are created and edited: flexmeasures add automation --unmonitored, flexmeasures edit automation --monitored/--unmonitored, the automations API (POST and PATCH), and a Monitored checkbox in the add and edit forms on the Automations page. Copying an automation keeps the original's choice.
  • Shown by flexmeasures show automations (a column, and a row in the detail view), by the API, and on the Automations page as a muted bell beside the status of an automation that is not monitored.
  • flexmeasures monitor automations leaves out the runs of unmonitored automations, through one filter in its query.

Look & Feel

An unmonitored automation gets a muted bell beside its status on the Automations page:

automations-table-unmonitored-bell

The add and edit forms have a Monitored checkbox under Active:

edit-automation-monitored-checkbox

flexmeasures show automations:

  ID  Asset                Name                            Type        Active    Monitored    Cron       Timezone
----  -------------------  ------------------------------  ----------  --------  -----------  ---------  ----------
   3  toy-battery (ID: 6)  Battery schedules               scheduling  yes       yes          * * * * *  UTC
   4  toy-battery (ID: 6)  Experimental battery schedules  scheduling  yes       no           * * * * *  UTC

With both automations failing, the digest only names the monitored one.
Once the experiment is monitored again, the next digest reports what it missed, collapsed:

Automation 4 'Experimental battery schedules' (scheduling) on asset 6 'toy-battery' of organisation 'Toy Account':
  failed 2 times (execution failed: 2), last at 2026-10-12 10:33 UTC (run 6)
  last error: UnknownPricesException: Prices unknown for planning window. (sensor 7)

Decisions worth a look

  • Unmonitored automations are left out of every digest, the host's included, so the filter sits once in the shared query.
  • Runs that fail while an automation is unmonitored are not marked as reported. Once it is monitored again, the next digest reports them, collapsed into one entry.
  • The field is set beside the type handler rather than by it, so no handler, a plugin's included, needs to know about monitoring.

How to test

Automated:

  • test_monitor_automations_fresh_db.py: an unmonitored automation's failed run is not reported, and is reported once the automation is monitored again.
  • test_automations_api.py: creating an unmonitored automation and patching it back to monitored. test_asset_copy_automations_fresh_db.py: a copy keeps the original's choice.
  • test_automations.py (CLI): add --unmonitored, the table, detail and JSON views, and edit --monitored.
  • test_automation_copy.py (JS): Copy carries the choice into the creation form, and a blank form is monitored again.
  • Each test was seen to fail by breaking what it covers.

Manually (needs Redis, and a mail catcher such as Mailpit; the digest is also logged):

  1. flexmeasures db upgrade; existing automations show Monitored: yes in flexmeasures show automations.
  2. flexmeasures add toy-account --kind battery. Its price sensor has no data, so schedules on the toy battery fail. Add two schedule automations there with --cron "* * * * *" --type scheduling --parameters schedule.yml, the second one with --unmonitored, where schedule.yml is:
    duration: PT12H
    flex-model:
      - sensor: <battery discharging sensor id>
        soc-min: 0 kWh
        soc-max: 450 kWh
        power-capacity: 500 kW
  3. flexmeasures jobs run-automations, then flexmeasures jobs run-worker --queue scheduling until both jobs have failed, then flexmeasures monitor automations: only the monitored automation is reported.
  4. Let both fail once more, then flexmeasures edit automation --id <experiment id> --monitored (or PATCH .../automations/<id> with {"monitored": true}), and run the monitor again: the experiment is reported once, as "failed 2 times". The asset's audit log says "monitoring switched on", and the automation's next run is unaffected.
  5. On the Automations page: the bell on the unmonitored row; Monitored ticked in a new form; the edit form showing and saving the current value; Copy of an unmonitored automation creating an unmonitored copy.

Related Items


Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

…s end

Context:
- Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs.

Change:
- record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded.
- The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job.
- Plugin jobs carry automation_run_id in their trigger.
- A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended.

Change:
- Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded.
- A plugin run records its job's outcome, also when the job finished before the run recorded it.
- Report run tests check the run's outcome: succeeded, nothing to do, and refused.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, as forecast runs did.

Change:
- Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, not only forecast runs.

Change:
- Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a monitor command will email a digest of failed automation runs,
  and has to report each failure once, also when a monitoring run is missed.

Change:
- Add a nullable alerted_at column to AutomationRun, with a migration.
- Add a partial index on the runs the monitor looks at
  (not reported yet, and failed, running or not fully dispatched),
  so that its hourly query does not scan the whole run history.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a run whose worker died stays "running" for good, so monitoring needs a threshold after which it counts as stuck.

Change:
- Add the setting, defaulting to six hours, well above any job timeout.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…rted yet

Context:
- #2519: monitoring reports runs whose execution failed, which are stuck, or whose dispatch failed for good.

Change:
- Add get_unalerted_failed_automation_runs, with one criterion for every automation type:
  only a forecast run with dispatch attempts to spare is left out, as it will be dispatched again,
  and so is a run that a runner still holds a live claim on.
- Add group_failed_runs_by_automation, collapsing failures per automation with their counts, latest run and latest error.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: automations fail silently unless someone watches their asset's Automations page.

Change:
- Email one digest of the failed runs to the monitoring recipients (or --recipient),
  collapsed per automation, and mark the runs as reported once it is sent.
- Send nothing when nothing failed, and refuse to run without recipients,
  rather than marking runs as reported that nobody heard of.
- --stuck-after-minutes overrides FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER.
- Kept out of flexmeasures jobs run-automations, so a mail problem never holds up dispatching.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: cover the new monitor command.

Change:
- Drive forecast runs to a failed execution, a stuck execution and an exhausted dispatch through the real runner and worker,
  and check each is reported once, and not before it counts.
- Check which runs count as failed for other types, the threshold's config and CLI override,
  collapsing per automation, and refusing to run without recipients.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: new monitor command and setting.

Change:
- Describe what the command reports in the automations feature docs, and point to it from the host's error monitoring page.
- List the command, document the stuck threshold setting, and add a CLI changelog entry.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Add a `monitored` flag to automations, true by default and for every existing automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…editing or copying it

A copy keeps the original's choice.
It is set beside the type handler rather than by it, so that no handler, a plugin's included, needs to know about monitoring.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…nd copying that choice

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
`flexmeasures add automation --unmonitored` and `flexmeasures edit automation --monitored/--unmonitored`,
with a Monitored column and row in `flexmeasures show automations`.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
A Monitored checkbox in the add and edit forms, carried over by Copy,
and a muted bell beside the activation status of an automation that is not monitored.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor.

Change:
- Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py.
- Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor.
- The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type.
- The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Builds the host digest on top of the run outcomes of schedule, report and plugin runs.
Brings in current main as well.
The only conflict was the CLI changelog, where both sides' entries are kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained migrations since this branch started, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise f7c2e8b1a409, main's head, rather than c5e1a7b94d20.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ng digest

Context:
- #2670 has schedule, report and plugin runs record how their jobs ended,
  so their failed executions can now reach the host digest.

Change:
- Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error.
- The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…xecution

Context:
- Since #2670, every automation type records how its run's jobs ended.

Change:
- Remove the note from the monitoring section of the automations docs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- PR #2671 adds flexmeasures monitor automations.

Change:
- Add one bullet for it to the per-PR automations list.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida
BelhsanHmida changed the base branch from main to feat/2519-monitor-automations October 12, 2026 01:46
…tomations-opt-out

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida and others added 9 commits October 12, 2026 02:52
Context:
- The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue.

Change:
- Say execution state and job records are kept for every type of automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py.

Change:
- Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
… them

A schedule run now names its jobs after itself and records them before they are queued,
so this branch no longer records them once they are queued, which would record each job twice.
Its jobs report how they ended through the same decorator as before, as they are looked up by their RQ ID.
A sequential schedule's device jobs are keyed by the sensor they schedule, rather than by their position.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…ts stored summary

Every queue's exception handler now stores the type and message of a job's error in its meta (#2698),
so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line.
The meta is read afresh, as the run holds the job as it was when queued.
A job without such a summary is still described by its traceback, without a type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…-monitor-automations

Prepares for merging #2670's squash commit, whose tree is identical to this tip.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Its content is already here from the branch's own lineage, so conflicts take this branch's side.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The only conflict was the CLI changelog, where main had moved the `show automations` entry up; this branch's `monitor automations` entry is kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained a merge migration since the last sync, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise bf22b21d0990, main's head, rather than f7c2e8b1a409.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

@Flix6x Flix6x left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Mohamed! We reviewed this together as a team today, on an isolated instance with your branch plus the current #2671 merged in (a clean merge). We went through the CLI, the API, the Automations page and the docs, using the toy battery from your test steps. Opting out works as described: the monitor leaves out the experiment's failed runs, the copy keeps the original's choice, and every switch lands in the audit log. Some changes before merging, and two decisions where we'd like to change the behaviour.

Needs a change

  1. Changelogs. The API changelog needs an entry for the new monitored field on the automation endpoints (POST, PATCH and the responses). The CLI changelog needs one for flexmeasures add automation --unmonitored, flexmeasures edit automation --monitored/--unmonitored, and the new column in flexmeasures show automations.
  2. Bring in #2671. The branch is 50 commits behind #2671, which has taken in main since. Merging it in is clean.
  3. Who may switch monitoring off. Today, any member of the asset's organisation may do it, even without any role, while the digest goes to the host's monitoring recipients. So the people who stop hearing about the failures aren't the ones who decide. For now, please allow only admins (and the host) to change monitored, in the API, and hide or disable the checkbox for others.
  4. The digest after monitoring is switched back on. We switched the experiment's monitoring off, let a month of failed runs pass (2,880 at every 15 minutes), and switched it back on. The next digest said "failed 2882 times", with only the last failure, and nothing showed that these runs failed while the automation was not monitored. Please name the first failure as well, and say that the runs failed while the automation was not monitored.

@nhoening wants to think about 3 and 4 some more, so please treat those as our current view rather than final.

Suggestions

  1. A filter on the Automations page: a toggle for monitored, unmonitored or all automations, with all as the default. The muted bell sits in the Active column, so sorting ignores it, and unmonitored automations are hard to find on an asset with many of them. The bell is also an icon with only a hover title, so screen readers may not announce it.
  2. Links in the digest to the asset's Automations page, and to the run's jobs and errors (the logs themselves aren't on the web). The digest is sent from the CLI, where FlexMeasures doesn't know its own web address, so this needs a setting for it (e.g. a base URL, or SERVER_NAME). It may fit #2671 better than this PR.

Open question, not for this PR alone

  1. #2856 (critical automations) adds a second flag, which calls host hooks when a run of a critical automation fails. With both flags, an automation can be critical and unmonitored. We still need to think about how the two relate, and whether they should become one setting. Until then, it would help if both PRs said in their docs what happens when both flags are set.

Not this PR: #2671's migration doesn't mark old failed runs as reported, so a host's first digest would report every run that ever failed.

@nhoening nhoening left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Allow only admins is the right choice in light of this PR. #2703 adds more recipients than just host-defined ones, but for now this still holds.

  2. I agree. Name the time of the first failuire, then the user can see the scope.

  3. We have --monitored/--unmonitored and --critical/--not-critical. For the time being, I believe they both have their distinct meaning. Indeed, when monitoring is off for an automation marked as cricital, I'd say let's keep it silent. But actually, right now we only call a hook governed by plugins. That might not be busy with monitoring but do other logic. #2856 should pass the monitored flag to the hook (building on this PR).

…or CLI

Requested in review, so other code can reuse it.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Otherwise a host's first digest lists every run that ever failed.
The upgrade marks the runs that the monitor would report now, by its rules at this revision, and leaves out runs that may still be retried or finish.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…tomations page

The digest is sent from the CLI, where FlexMeasures does not know its own address, so the new FLEXMEASURES_BASE_URL setting gives it.
Without the setting, the digest carries no links, as before.
The Automations page opens the tab and info panel of the automation named in ?automation=<id>, which show its latest run, its error and its jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…utomations-opt-out

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	flexmeasures/cli/tests/test_monitor_automations_fresh_db.py
Switching monitoring off silences the digest for whoever receives it, so it is up to the host's admins and the account-admins of the asset's organisation, rather than to everyone who may manage the automation.
The API refuses others with a 403 when the request would change the monitoring, and the Automations page disables the checkbox for them and leaves the field out of what it sends.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ored

Once monitoring is switched back on, the digest reports the runs that failed meanwhile, which can be thousands.
It now names the first failure besides the last one, and says how many runs failed before monitoring was switched back on, and when that was.
The automation records that moment in a new monitoring_resumed_at column.
The docs also say who may switch monitoring, and that it only affects the digest.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ion form

The page script now reads user_can_change_monitoring, so the test helpers render it, and the creation form is rendered through Jinja rather than patched by hand.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Unmonitored automations are hard to find on an asset with many automations, so a filter lists all, only monitored or only unmonitored ones, all by default.
The Active column also sorts unmonitored automations together, and the muted bell carries text for screen readers.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
… change logs

Requested in review.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
…utomations-opt-out

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/api/change_log.rst
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
@BelhsanHmida
BelhsanHmida requested a review from Flix6x November 16, 2026 00:30
@BelhsanHmida

Copy link
Copy Markdown
Contributor Author

1, 2: changelogs in 90d76ef, #2671 merged in d27047c.
3: only host admins and the asset's account-admins can switch monitoring; others get a 403 and a disabled checkbox (bc49707, 67f40a0).
4: the digest names the first failure and how many runs failed while unmonitored (dbca642). 5: filter in 54789ab. 6: links went into #2671 (644b604).
7: the docs say switching it off only affects the digest; #2856 can pass monitored to its hook as suggested

Base automatically changed from feat/2519-monitor-automations to main November 17, 2026 22:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let an automation opt out of monitoring

3 participants